Description

New test generation strategy is:

• If function is fully annotated and type inference algorithm cannot infer type we use all time from UI-settings to fuzzing and test generation;
• If type inference and infer type we try to fuzz tests for one annotation with the following restrictions:
  • max 150 successfully executable tests (with or without exeception)
  • max 10 invalid tests (when we generate incorrect arguments and have TypeError or AttributeError; or we cannot generate any valid representative of the inferred type)
  • and 5 additional tests (as a precaution)
  • until time limit

In future we can modify strategies and add it to UI. For example, wait max coverage.

Fixes #1874
Fixes #1831

How to test

Manual tests

Fully annotated function

For example,

def div(a: int, b: int) :
    return a / b

Expected: two tests (valid and with ZeroDivisionError), test generation process lasted all time from UI-settings

Or,

def str_test(x: str):
    if x[0] == 'b':
        if x[1] == 'a':
            if x[2] == 'd':
                return 'Very bad!'

Expected: all time generation, one test with x = 'ba' or longer (with timeout 60s) and all covered branches with timeout 180s

Without or partially annotated function

For example,

def str_test(x):
    if x[0] == 'b':
        if x[1] == 'a':
            if x[2] == 'd':
                return 'Very bad!'

Expected: all time generation, one test with x = 'b', no one with 'ba' and longer (regardless of timeout)

Self-check list

Check off the item if the statement is true. Hint: [x] is a marked item.

Please do not delete the list or its items.

• I've set the proper labels for my PR (at least, for category and component).
• PR title and description are clear and intelligible.
• I've added enough comments to my code, particularly in hard-to-understand areas.
• The functionality I've repaired, changed or added is covered with automated tests.
• Manual tests have been provided optionally.
• The documentation for the functionality I've been working on is up-to-date.